Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapr deprecation #441

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Mapr deprecation #441

wants to merge 13 commits into from

Conversation

will-moore
Copy link
Member

@will-moore will-moore commented Nov 15, 2024

This represents the changes I have been making manually on idr-testing with the replacement of mapr searches with searchengine, described at IDR/idr-gallery#45

Summary of changes:

  • Move mapr app AFTER idr-gallery so that URLs such as /mapr/gene/ get handled by idr-gallery instead of mapr
  • Remove the mapr cache so that we don't get the cached response for these URLs
  • For /mapr/api/[key]/ requests we now response with a 410 response. [EDIT] Now redirects to /searchengine/apidocs/ 302 response - see below.
  • Bumps idr-gallery to 3.12.0 to include the 'mapr-redirect' PR above and Fix autocomplete studies idr-gallery#47.

NB: I couldn't see where to add the file I created with:

sudo vi /usr/share/nginx/html/410.html (based on the 50x.html)

<!DOCTYPE html>
<html>
<head>
<title>Error</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<p>mapr/api pages have been removed.</p>
<p>
  Please use searchengine instead. See
  <a href="https://idr.openmicroscopy.org/searchengine/apidocs/">https://idr.openmicroscopy.org/searchengine/apidocs/</a>
</p>
</body>
</html>

Also the config for this file looks like:

    error_page 410 /410.html;
    location = /410.html {
        root   /usr/share/nginx/html;
    }

In my testing, I had added this in proxy-default.conf right before the location entry that I have added above:

    location ~ ^/mapr/api/(antibody|gene|compound|sirna|omap|phenotype|organism|orf|cellline|protein)($|/) {
        return 410;
    }

The error_page 410 about follows on from error_page 50x etc section in the same file, but I don't see that this is specified anywhere in this repo, so maybe that comes from existing nginx config somewhere?

@will-moore
Copy link
Member Author

Build failures:

WARNING  Listing 3 violation(s) that are fatal
syntax-check[specific]: couldn't resolve module/action 'os_server_actions'. This often indicates a misspelling, missing collection, or incorrect module path.
decommission/archive-instance-networks.yml:35:5

syntax-check[specific]: The field 'hosts' has an invalid value, which includes an undefined variable. The error was: 'idr_environment' is undefined. 'idr_environment' is undefined
decommission/archive-instance-services.yml:3:3

syntax-check[specific]: couldn't resolve module/action 'letsencrypt'. This often indicates a misspelling, missing collection, or incorrect module path.
idr-proxy-letsencrypt.yml:43:5

@sbesson
Copy link
Member

sbesson commented Nov 15, 2024

I believe the primary failure is actually during the converge step

  TASK [ome.iptables_raw : iptables-raw | enable iptables] ***********************
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  fatal: [idr-omeroreadonly-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}
  fatal: [idr-database-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}
  fatal: [idr-omeroreadwrite-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}

#440 contains the proposal to address this issue. Re-expressing the next steps that need to happen:

1- ome/ansible-role-iptables-raw#13 needs to be reviewed, merged and released
2- #440 needs to be updated with the new role version, reviewed and merged
3- origin/master can be merged into this branch to include the fix

@will-moore
Copy link
Member Author

will-moore commented Nov 18, 2024

@will-moore
Copy link
Member Author

As discussed this morning in answer to my question above about where to put:

    error_page 410 /410.html;
    location = /410.html {
        root   /usr/share/nginx/html;
    }

It seems that the equivalent 50x handling is part of this ansible role:

https://github.com/ome/ansible-role-nginx-proxy/blob/1a2eb446951fe831126128c747beb6221d142047/templates/nginx-confd-proxy.j2#L50

However, it's not clear how/where to add the custom /usr/share/nginx/html/410.html I had above.

This is somewhat similar to the 'maintenance page' we have specified at
https://github.com/IDR/idr.openmicroscopy.org/blob/master/omero-maintenance.html
https://idr.openmicroscopy.org/about/omero-maintenance.html

However, to show a similar page for the /mapr/api/ urls would require a redirect, since we're returning a html page that is not from within nginx (as the 50x.html page is) but is hosted at a different URL.

So, the options are:

    1. we simply redirect to /searchengine/apidocs/ from all /mapr/api/ pages. Simple config in IDR/deployment
    1. we leave this PR as it is, and we get a default nginx 410 response (simply says 410 Gone).
    1. We decide on a location to store a custom 410.html, update https://github.com/ome/ansible-role-nginx-proxy/ to write that file to that location, then update deployment to use that role. Problems are that this role is not only used by IDR, and that I don't know how to make those changes!

So, probably best to go for option 1. This may not be precisely the correct response, but it will be most user-friendly and easiest for us, bearing in mind it would be nice to release this next week or soon after.

@will-moore
Copy link
Member Author

Build is green now 👍 .
@dominikl will you be able to deploy this on a pilot so we can test? Then I can also deploy my idr-gallery changes on the same pilot and we can test with/without the idr-gallery changes. Thx

@sbesson
Copy link
Member

sbesson commented Nov 20, 2024

@will-moore with #440 in, can you merge origin/master into your branch to fix the conflict?

@will-moore
Copy link
Member Author

@sbesson Done

@sbesson
Copy link
Member

sbesson commented Nov 21, 2024

Deployed to test125 together with #442

TASK [ome.nginx_proxy : nginx | proxy cache config] ****************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-cache.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmp3t9v16lg/nginx-confd-proxy-cache.j2
@@ -5,7 +5,6 @@
 proxy_cache_path /var/cache/nginx/omerorender levels=1:2 keys_zone=omerorender:35m max_size=40g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omerometadata levels=1:2 keys_zone=omerometadata:65m max_size=25g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omeroapi levels=1:2 keys_zone=omeroapi:320m max_size=10g inactive=180d use_temp_path=off;
-proxy_cache_path /var/cache/nginx/omeromapr levels=1:2 keys_zone=omeromapr:100m max_size=5g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omeroviewers levels=1:2 keys_zone=omeroviewers:1m max_size=1g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/grafana levels=1:2 keys_zone=grafana:1m max_size=100m inactive=1m use_temp_path=off;
 
@@ -18,7 +17,6 @@
     "~web(client|gateway)/get_thumbnail*" 0;
     "~(webclient/)?api/*" 0;
     "~static/*" 0;
-    "~mapr/*" 0;
     "~gallery-api/*" 0;
     "~webclient/img_detail/*" 0;
     "~iviewer/*" 0;
@@ -46,7 +44,6 @@
     "~web(client|gateway)/render_*/*" omerorender;
     "~webclient/metadata_*" omerometadata;
     "~(webclient/)?api/*" omeroapi;
-    # "~mapr/*" omeromapr;
     "~webclient/img_detail/*" omeroviewers;
     "~iviewer/*" omeroviewers;
     "~grafana/*" grafana;

changed: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy cache config] ****************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | check websockets configuration] ****************************************************************************************************************
skipping: [test125-proxy] => (item={'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*', 'server': 'http://omeroreadonly', 'cache_validity': '1d', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omeromapr', 'location': '~ /mapr/*', 'server': 'http://omeroreadonly', 'cache_validity': '180d', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadonly', 'cache_validity': '1d'}) 
skipping: [test125-proxy] => (item={'name': 'omero', 'location': '/', 'server': 'http://omeroreadonly', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omerowebsockets', 'location': '= /omero-ws', 'server': 'http://omeroreadonlywebsockets', 'websockets': True, 'websocketsonly': True, 'read_timeout': 86400}) 
skipping: [test125-proxy] => (item={'name': 'grafanarender', 'location': '^~ /grafana/', 'server': 'http://192.168.2.92:3000/render/dashboard/db/', 'cache_validity': '1m'}) 
skipping: [test125-proxy] => (item={'name': 'prometheusfederate', 'location': '^~ /prometheus/federate', 'server': 'http://192.168.2.92:9090/federate', 'cache_validity': '15s'}) 
skipping: [test125-proxy] => (item={'name': 'searchengine', 'location': '^~ /searchengine', 'server': 'http://192.168.2.197:5577/searchengine', 'host_header': '$host/searchengine'}) 

TASK [ome.nginx_proxy : nginx | proxy websockets config] ***********************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy websockets config] ***********************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | check redirect configuration] ******************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy redirects config] ************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-redirect.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpjl5hj5mc/nginx-confd-proxy-redirect.j2
@@ -14,8 +14,4 @@
     ~/pgpc /webclient/?show=screen-1151;
     /study/idr0083/figure/4i /webclient/img_detail/9822151/?dataset=10201&x=34105&y=84808&zm=25&c=1|318:9927$808080&m=g;
     /study/idr0083/figure/4r /webclient/img_detail/9822152/?dataset=10201&x=80560&y=77440&zm=66&c=1|495:9204$808080&m=g;
-    /mapr/api/antibody /searchengine/apidocs/;
-    /mapr/api/compound /searchengine/apidocs/;
-    /mapr/api/gene /searchengine/apidocs/;
-    /mapr/api/config /mapr/api/config;
 }

changed: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy redirects config] ************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream-conf.d directory] ***********************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream proxy config] ***************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream proxy config] ***************************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy config] **********************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpnatinmsu/nginx-confd-proxy.j2
@@ -12,7 +12,9 @@
 
     ssl_certificate /etc/ssl/localcerts/bundled.crt;
     ssl_certificate_key /etc/ssl/localcerts/server.key;
-    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  # don't use SSLv3 ref: POODLE
+    ssl_protocols  TLSv1.2 TLSv1.3;  # don't use SSLv3 ref: POODLE
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
+
 
     #charset koi8-r;
     #access_log  /var/log/nginx/log/host.access.log  main;
@@ -44,9 +46,8 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    
     location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 /searchengine/apidocs/;
+        return 302 $redirect_uri;
     }
 
     location = /webclient {

changed: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']})
--- before: /etc/nginx/conf.d/proxy-cachebuster.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmp2pvaxxro/nginx-confd-proxy.j2
@@ -33,6 +33,9 @@
     port_in_redirect off;
 
     location ~ ^/(mito|tara|pgpc|study)($|/) {
+        return 302 $redirect_uri;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
         return 302 $redirect_uri;
     }
 

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]})
--- before: /etc/nginx/conf.d/proxy-idr-demo.openmicroscopy.org.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpifpytgda/nginx-confd-proxy.j2
@@ -10,7 +10,9 @@
 
     ssl_certificate /etc/ssl/localcerts/bundled.crt;
     ssl_certificate_key /etc/ssl/localcerts/server.key;
-    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  # don't use SSLv3 ref: POODLE
+    ssl_protocols  TLSv1.2 TLSv1.3;  # don't use SSLv3 ref: POODLE
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
+
 
     #charset koi8-r;
     #access_log  /var/log/nginx/log/host.access.log  main;

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []})

@will-moore
Copy link
Member Author

@sbesson - mostly that diff looks as expected (but not at

     }
-    
     location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 /searchengine/apidocs/;
+        return 302 $redirect_uri;
     }

I had manually added this myself by editing of /etc/nginx/conf.d/proxy-default.conf:

     location ~ ^/mapr/api/(?!config).*($|/) {
        return 302 /searchengine/apidocs/;
     }

so it looks like the diff is comparing my manual changes here to the changes after your update?
With the change there return 302 $redirect_uri; this is failing to handle redirects e.g. for /mapr/api/gene/ because there's no entries for them in the proxy-redirect.conf.

But I also don't understand in proxy-redirect.conf we have this diff:

-    /mapr/api/antibody /searchengine/apidocs/;
-    /mapr/api/compound /searchengine/apidocs/;
-    /mapr/api/gene /searchengine/apidocs/;
-    /mapr/api/config /mapr/api/config;

This looks like removal of redirects to /searchengine/apidocs/. I didn't add these manually so I don't know where these came from?

@sbesson
Copy link
Member

sbesson commented Nov 21, 2024

so it looks like the diff is comparing my manual changes here to the changes after your update?

That's correct, this is capturing the changes was applied to test125 from the state it was previously including local modifications.

This looks like removal of redirects to /searchengine/apidocs/. I didn't add these manually so I don't know where these came from?

Clearly someone edited this file as these changes are neither on prod124 or prod125.

If you don't want to use the redirect map, then you probably need to put your 302 redirect under nginx_proxy_direct_locations rather than nginx_proxy_redirect_map_locations

@will-moore
Copy link
Member Author

Ah - understood! Done, thanks.

@sbesson
Copy link
Member

sbesson commented Nov 21, 2024

After running the idr-proxy.yml playbook with the last commit

TASK [ome.nginx_proxy : nginx | proxy config] **************************************************************************************************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-67117nlkwjgz4/tmp4odhrluu/nginx-confd-proxy.j2
@@ -46,9 +46,6 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 $redirect_uri;
-    }
 
     location = /webclient {
         return 302 /webclient/?experimenter=-1;
@@ -76,6 +73,9 @@
     }
     location = /submitter-survey {
         return 302 https://forms.gle/beR4vqLrgLgRzgSX9;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
+        return 302 /searchengine/apidocs/;
     }
 
     location ~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/* {

changed: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']})
--- before: /etc/nginx/conf.d/proxy-cachebuster.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-67117nlkwjgz4/tmp4y0krw4k/nginx-confd-proxy.j2
@@ -35,9 +35,6 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 $redirect_uri;
-    }
 
     location = /webclient {
         return 302 /webclient/?experimenter=-1;
@@ -65,6 +62,9 @@
     }
     location = /submitter-survey {
         return 302 https://forms.gle/beR4vqLrgLgRzgSX9;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
+        return 302 /searchengine/apidocs/;
     }
 
     location ~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/* {

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]})
ok: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []})

@will-moore
Copy link
Member Author

@sbesson Thanks - that looks like it's working fine now 👍 .

@sbesson
Copy link
Member

sbesson commented Nov 21, 2024

@sbesson Thanks - that looks like it's working fine now 👍 .

👍 Do I understand correctly that this PR is now only waiting on a release of the idr-gallery application and the corresponding update to

- idr-gallery==3.11.0
?

@will-moore
Copy link
Member Author

Yes, I think so

@will-moore
Copy link
Member Author

I'll wait for idr-gallery release (and mapr too) then bump them both in this PR.

@will-moore will-moore mentioned this pull request Nov 25, 2024
sbesson
sbesson previously approved these changes Nov 27, 2024
Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The components and NGINX changes are consistent with the discussions around the deprecation of the mapr endpoints from the last few weeks and have been successfully deployed at test125.

TASK [ome.omero_web : omero web | install requirements] *************************************************************************************************************************************************
changed: [test125-omeroreadonly-2]
changed: [test125-omeroreadonly-1]
changed: [test125-omeroreadonly-4]
changed: [test125-omeroreadwrite]
changed: [test125-omeroreadonly-3]
...
TASK [ome.omero_web : omero web | flush systemd handlers] ***********************************************************************************************************************************************

RUNNING HANDLER [ome.omero_web : omero-web rewrite omero-web configuration] *****************************************************************************************************************************
changed: [test125-omeroreadonly-1]
changed: [test125-omeroreadonly-3]
changed: [test125-omeroreadonly-4]
changed: [test125-omeroreadwrite]
changed: [test125-omeroreadonly-2]

RUNNING HANDLER [ome.omero_web : omero-web restart omero-web] *******************************************************************************************************************************************
changed: [test125-omeroreadonly-1]
changed: [test125-omeroreadonly-4]
changed: [test125-omeroreadwrite]
changed: [test125-omeroreadonly-2]
changed: [test125-omeroreadonly-3]

Approving from a deployment perspective and assigning a few reviewers for the functional testing before we apply this on prod125

@will-moore
Copy link
Member Author

Testing on idr-testing looks good for me. Search, redirects etc.

@francesw
Copy link
Member

Tested on idr-testing. Looks good. Search, redirects and pop-up working as expected.

@pwalczysko
Copy link
Contributor

Clicking on Homo sapiens leads to the old mapr page on idr-testing

@will-moore
Copy link
Member Author

Still need to find why
https://idr-testing.openmicroscopy.org/mapr/organism/?value=Homo%20sapiens
is being cached instead of redirecting to idr-gallery etc.

Looking for "sapiens" etc in ngninx conf finds nothing. Looking for mapr finds some things...

[wmoore@test125-proxy ~]$ grep -r sap /etc/nginx/
[wmoore@test125-proxy ~]$ grep -r mapr /etc/nginx/
/etc/nginx/conf.d/proxy-cachebuster.conf:    location ~ ^/mapr/api/(?!config).*($|/) {
/etc/nginx/conf.d/proxy-cachebuster.conf:    location ~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/* {
/etc/nginx/conf.d/proxy-default.conf:    location ~ ^/mapr/api/(?!config).*($|/) {
/etc/nginx/conf.d/proxy-default.conf:    location ~ /mapr/* {
/etc/nginx/conf.d/proxy-gone.conf:    location ~ ^/(mapr/api/compound|mapr/api/antibody)($|/) {

The cache itself is stored at /var/cache/nginx

Try

[wmoore@test125-proxy ~]$ sudo mv /var/cache/nginx/omeromapr /var/cache/nginx/omeromapr2
$ sudo systemctl restart nginx

Now, this redirects correctly...
http://localhost:9000/search/?key=Organism&value=Homo+sapiens&operator=equals

Try to find weblient page with "Homo sapiens" in caches. Use icon_thumbnails_template as a token that is found in the html from webclient (and mapr). sapiens is only found in https://idr-testing.openmicroscopy.org/mapr/organism/

Found 1 under /var/cache/nginx/omero. Renamed but made no difference.

[wmoore@test125-proxy ~]$ sudo grep -r icon_thumbnails_template /var/cache/nginx/omero 
grep: /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90: binary file matches
[wmoore@test125-proxy ~]$ sudo mv /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90 /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90_delete
$ sudo systemctl restart nginx

Other dirs either don't find anything or take too long to grep:

[wmoore@test125-proxy ~]$ sudo ls /var/cache/nginx/
client_temp  fastcgi_temp  grafana  omero  omeroapi  omeromapr2  omerometadata	omerorender  omerostatic  omerothumbnail  omeroviewers	proxy_temp  scgi_temp  uwsgi_temp

@will-moore
Copy link
Member Author

will-moore commented Dec 2, 2024

Ah-ha! After waiting a couple of hours, https://idr-testing.openmicroscopy.org/mapr/organism/?value=Homo%20sapiens has stopped returning a cached page and now re-directs to idr-gallery search as intended!

NB: $ sudo grep -r icon_thumbnails_template /var/cache/nginx/omero now returns NOTHING - the cache I renamed above to /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90_delete has now disappeared (expired)?

Let's test some more on idr-testing with the omeromapr cache renamed (disabled) as above to check that it really isn't being used now...

EDIT: Done a fair bit of testing and everything seems to be working, apart from a couple of "no results" - issue created at ome/omero_search_engine#104

@will-moore
Copy link
Member Author

@pwalczysko Can you confirm that clicking on Homo sapiens now redirects to `/search/ on IDR testing?

@pwalczysko
Copy link
Contributor

@pwalczysko Can you confirm that clicking on Homo sapiens now redirects to `/search/ on IDR testing?

Confirmed. FF on Mac M1, new private window. Click on Homo sapiens goes to search https://idr-testing.openmicroscopy.org/search/?key=Organism&value=Homo+sapiens&operator=equals

Copy link
Contributor

@pwalczysko pwalczysko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some more extensive testing on idr-testing just now. LGTM

@sbesson
Copy link
Member

sbesson commented Dec 2, 2024

From today's conversation in OME Slack

willmoore11:15 AM
OK if I try something like this
[wmoore@test125-proxy ~]$ sudo mv /var/cache/nginx/omeromapr /var/cache/nginx/omeromapr2
?
willmoore 12:09 PM
Hmmm - no joy there. Made no difference. Does nginx logging capture cache handling? Where would I look for those?
willmoore 12:34 PM
Otherwise I'm kinda clueless what/where to dig next...
seb 1:16 PM
@ Will moore try now?
I commented out all the `proxy_cache entries under the location ~ /mapr/* block and restarted NGINX. If that's successful for you guys too, that's probably an indication there is still some caching and this block must go away (edited)

Were the NGINX changes applied to test125 reverted prior to the testing above? Or should the playbook be updated to remove the mapr block as there is no more use of the cache?

- name: omeromapr
location: ~ /mapr/*
server: http://omeroreadonly
cache_validity: 180d
maintenance_flag: "{{ omero_maintenance_flag }}"
maintenance_uri: "{{ omero_maintenance_uri }}"

@will-moore
Copy link
Member Author

Were the NGINX changes applied to test125 reverted prior to the testing above?

No.

Or should the playbook be updated to remove the mapr block as there is no more use of the cache?

I don't know about any playbook, but sounds like a yes. Anyway the mapr caching should already be disabled by this change in this PR:

nginx_proxy_cache_match_uri:
#- '"~webclient/api/paths_to_object*"'
- '"~web(client|gateway)/(metadata|render)_*"'
- '"~web(client|gateway)/get_thumbnail*"'
- '"~(webclient/)?api/*"'
- '"~static/*"'
# - '"~mapr/*"'

@sbesson
Copy link
Member

sbesson commented Dec 3, 2024

Anyway the mapr caching should already be disabled by this change in this PR:

These changes have been deployed since last week (#441 (review)). My understanding is that we had clearly established that these changes alone were not sufficient to since the Homo sapiens mapr URLs were still not redirecting to the search endpoint at yesterday's IDR meeting.

Following the Monday discussion, two changes were performed on test125:

  • the omeromapr NGINX cache was renamed as omeromapr2
  • the proxy-default.conf NGINX configuration was modified with the following diff compared to production
113,119c118,124
<         proxy_cache            $cache_zone_name;
<         proxy_cache_key        $cache_key;
<         proxy_cache_valid      200 180d;
<         proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
<         proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
<         proxy_cache_background_update on;
<         proxy_cache_bypass     $cache_refresh;
---
>         #proxy_cache            $cache_zone_name;
>         #proxy_cache_key        $cache_key;
>         #proxy_cache_valid      200 180d;
>         #proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
>         #proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
>         #proxy_cache_background_update on;
>         #proxy_cache_bypass     $cache_refresh;

I just reverted the latter configuration changes and restarted NGINX on test125 and the redirect of the /mapr/organism/?value=Homo%20sapiens endpoint are no longer working for me. Can you guys confirm it's the same on your end?

@will-moore
Copy link
Member Author

will-moore commented Dec 3, 2024

There was also a single cache that showed up in my grepping for home page html that I renamed (see above):

sudo mv /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90 /var/cache/nginx/omero/0/c9/1a0528c9931482c19788252d1cb13c90_delete

This could have been cached in some custom step somewhere and might be responsible for the custom behaviour of Homo sapiens?

I can confirm that /mapr/organism/?value=Homo%20sapiens is no-longer redirecting for me on idr-testing.

EDIT: ...and also I'm not seeing anything matching that page under omero cache:

[wmoore@test125-proxy ~]$ sudo grep -r icon_thumbnails_template /var/cache/nginx/omero
[wmoore@test125-proxy ~]$

@will-moore
Copy link
Member Author

@sbesson when you say "two changes were performed on test125", I understand the first (which I did) but not:

the proxy-default.conf NGINX configuration was modified with the following diff compared to production

Do you know what caused that? I don't think I did that (unless by some mechanism I don't understand)?!
We clearly need that change. But it's not specified in this PR, right? Is it specified in some other playbook PR?

@sbesson
Copy link
Member

sbesson commented Dec 3, 2024

Do you know what caused that? I don't think I did that (unless by some mechanism I don't understand)?!

Apologies for the confusion, clearly the communication did not go through. I applied this change manually on test125 after you indicated that you were running out of ideas on what to try next.

But it's not specified in this PR, right? Is it specified in some other playbook PR?

Absolutely, this change was applied manually on top of the configuration set by the state of this PR so we need another change in addition to what is suggested here. As mentioned in #441 (comment), my suspicion is that we want to remove

- name: omeromapr
location: ~ /mapr/*
server: http://omeroreadonly
cache_validity: 180d
maintenance_flag: "{{ omero_maintenance_flag }}"
maintenance_uri: "{{ omero_maintenance_uri }}"
. Should I try this in dry-run mode first and see the outcome?

@will-moore
Copy link
Member Author

Yes please! Go ahead, thanks.

@sbesson
Copy link
Member

sbesson commented Dec 3, 2024

With

diff --git a/ansible/group_vars/proxy-hosts.yml b/ansible/group_vars/proxy-hosts.yml
index 8a6cd5d..d46d9fc 100644
--- a/ansible/group_vars/proxy-hosts.yml
+++ b/ansible/group_vars/proxy-hosts.yml
@@ -64,12 +64,6 @@ _nginx_proxy_backends_omero:
   cache_validity: 1d
   maintenance_flag: "{{ omero_maintenance_flag }}"
   maintenance_uri: "{{ omero_maintenance_uri }}"
-- name: omeromapr
-  location: ~ /mapr/*
-  server: http://omeroreadonly
-  cache_validity: 180d
-  maintenance_flag: "{{ omero_maintenance_flag }}"
-  maintenance_uri: "{{ omero_maintenance_uri }}"
 - name: omerostatic
   location: ~ /static/*
   server: http://omeroreadonly

I have the following changes in dry-run mode

TASK [ome.nginx_proxy : nginx | proxy config] ***********************************************************************************************************************************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-6847698rc123m/tmpzfgu325k/nginx-confd-proxy.j2
@@ -86,38 +86,6 @@
         proxy_cache            $cache_zone_name;
         proxy_cache_key        $cache_key;
         proxy_cache_valid      200 1d;
-        proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
-        proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
-        proxy_cache_background_update on;
-        proxy_cache_bypass     $cache_refresh;
-        proxy_no_cache         $skip_cache;
-
-
-        proxy_cache_lock          on;
-        proxy_cache_lock_age      1m;
-        proxy_cache_lock_timeout  1m;
-
-        proxy_ignore_headers   "Set-Cookie" "Vary" "Expires";
-        proxy_hide_header Set-Cookie;
-
-
-
-        if (-f /srv/www/omero-maintenance.flag) {
-            return 503;
-        }
-        error_page 503 /about/omero-maintenance.html;
-
-
-    }
-
-    location ~ /mapr/* {
-
-        proxy_pass http://omeroreadonly;
-        proxy_redirect http://omeroreadonly $scheme://$server_name;
-
-        proxy_cache            $cache_zone_name;
-        proxy_cache_key        $cache_key;
-        proxy_cache_valid      200 180d;
         proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
         proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
         proxy_cache_background_update on;

Should I try and execute this outside check-mode?

@will-moore
Copy link
Member Author

@sbesson - yes, great. Let's give it a test, thanks

@sbesson
Copy link
Member

sbesson commented Dec 3, 2024

The change above is now deployed on test125 and NGINX has been restarted by the playbook

TASK [ome.nginx_proxy : nginx | proxy config] ***********************************************************************************************************************************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-70170hrpvvmze/tmp4v98_6zl/nginx-confd-proxy.j2
@@ -86,38 +86,6 @@
         proxy_cache            $cache_zone_name;
         proxy_cache_key        $cache_key;
         proxy_cache_valid      200 1d;
-        proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
-        proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
-        proxy_cache_background_update on;
-        proxy_cache_bypass     $cache_refresh;
-        proxy_no_cache         $skip_cache;
-
-
-        proxy_cache_lock          on;
-        proxy_cache_lock_age      1m;
-        proxy_cache_lock_timeout  1m;
-
-        proxy_ignore_headers   "Set-Cookie" "Vary" "Expires";
-        proxy_hide_header Set-Cookie;
-
-
-
-        if (-f /srv/www/omero-maintenance.flag) {
-            return 503;
-        }
-        error_page 503 /about/omero-maintenance.html;
-
-
-    }
-
-    location ~ /mapr/* {
-
-        proxy_pass http://omeroreadonly;
-        proxy_redirect http://omeroreadonly $scheme://$server_name;
-
-        proxy_cache            $cache_zone_name;
-        proxy_cache_key        $cache_key;
-        proxy_cache_valid      200 180d;
         proxy_cache_methods    GET HEAD; # Only GET and HEAD methods apply
         proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
         proxy_cache_background_update on;

changed: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']})
ok: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]})
ok: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []})

TASK [ome.nginx_proxy : nginx | proxy upstream servers] *************************************************************************************************************************************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | create proxy ssl certificate directory] *********************************************************************************************************************************************************************************************************************
ok: [test125-proxy] => (item={'nginx_proxy_ssl_certificate': '/etc/ssl/localcerts/bundled.crt'})
skipping: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']}) 
skipping: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]}) 
skipping: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []}) 

TASK [ome.nginx_proxy : nginx | create proxy ssl certificate key directory] *****************************************************************************************************************************************************************************************************************
ok: [test125-proxy] => (item={'nginx_proxy_ssl_certificate_key': '/etc/ssl/localcerts/server.key'})
skipping: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']}) 
skipping: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]}) 
skipping: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []}) 

TASK [ome.nginx_proxy : nginx | copy proxy ssl certificate] *********************************************************************************************************************************************************************************************************************************
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | copy proxy ssl certificate key] *****************************************************************************************************************************************************************************************************************************
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy] => (item=None) 
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | start service] **********************************************************************************************************************************************************************************************************************************************
ok: [test125-proxy]

RUNNING HANDLER [ome.nginx : restart nginx] *************************************************************************************************************************************************************************************************************************************************
changed: [test125-proxy]

PLAY RECAP **********************************************************************************************************************************************************************************************************************************************************************************
test125-management         : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-omeroreadonly-1    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-omeroreadonly-2    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-omeroreadonly-3    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-omeroreadonly-4    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-omeroreadwrite     : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test125-proxy              : ok=44   changed=2    unreachable=0    failed=0    skipped=17   rescued=0    ignored=0   
test125-searchengine       : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

@will-moore
Copy link
Member Author

Bit of testing this morning - all seems to be working fine 👍 - no caching issues etc.

@sbesson sbesson dismissed their stale review December 4, 2024 08:52

Additional changes required - see #441 (comment)

@will-moore
Copy link
Member Author

That's the change above at #441 (comment) included in the last commit.

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the state of the test125 deployment. Once the functional behavior has been validated, this should be ready to deploy onto prod125.

@will-moore
Copy link
Member Author

@jburel could you have a look at this (it's deployed on idr-testing) as we'd like to get this merged at the end of this week or start of next week (I'm off tomorrow and Friday). Thanks!

- idr-gallery==3.11.0
- omero-mapr==0.5.3
- omero-iviewer==0.15.0
- idr-gallery==3.12.1
- omero-figure==6.2.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Figure is not used but some libraries bumped happened since and it could potential lead to some dependencies issues down the line

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see one of three options:

  • do nothing and capture as an issue
  • bump OMERO.figure - if so to which version?
  • remove OMERO.figure
    Which one would you recommend?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants